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ABSTRACT 

Local and regional advisory councils can provide; 
vocational programs with unique and essential' evaluation information. 
Most advisory council;^ a:re asked to review components of vocational 
programs about which they have limited experience and/or knowledge^ 
such as philosophy, objectives, and management. There a^e' several, 
often overlooked areas in which the expertise and consequent advice 
of the advisory council would be most helpful and appropriate. The 
Kentucky Advisory Committee Evaluation System (RACES) was designed to, 
tap the resource potential of the advisory council. KACES is a 
four-step eraluation which includes orientation, investigation, 
interpretat/on, and recommendation. The evaluation is 4)ased on two 
forms utilized by the advisory councils at the State, regional, and 
local levels. Reactions to the KACES program were very positive,- and 
recommendations about the KACES program were based on the pilot, 
testing and revisions and include points such as clearly defining the 
vocational teacher • s. role in evaluation, c[nd developing alternative 
interviewing times and techniques. (TM) , . 
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FOREWORD 

■ . 

We are pleased to present this, report to the Kentucky 
State' Advisory Council for Vocational Education and Manpower 
Development and Training. This report describes the sequence 
*of activities accomplished in developing and trying out the 
Kentucky Advisory Committee Evaluation System (KACES) . 

The Center would like to commend Mr. Billy Howard and the 
Kentucky State Advisory Council for Vocational Education ^ 
and Manpower Development and Training for' initiating a systematic 
effort to ^.ncrease the involvement of advisory committees 
in evaluating ,vocatiorial programs. Advisory committees are . 
a valuable resource for vocational 'education. Efforts to 
use advisory committees' time wisely and their eocpertise 
productively are highly commendable. 

Special acknowledgements ar^ extended to regional vocational 
education personnel and advisory committee members in Region 
II and R^ion VII of Kentucky who served on the project advisory 
comiTiittee- Participants on the project advisory committee 
included Mr- Biil/Hatley, Mr. John Gray, Mr. Rodney Dempsey, 
and Mrs. Pat Curtis from Region II and Mr^ James^ Patton\ Mrs. 
Dorothy Corday/ Mr. George Graham, Mr. Garry Haake, Mr. 
Joseph Flynn, Mr. Frank B. Raine, Mr. Don Brandt, and Dr. 
Dennis Savell from Region VII. 

Appreciation is also extfended to the local craft committee 
members, vocational teachers and regional staff who participated 
in the pilot test 'Of KACES. 

Sincere thanks are extended to Dr. N. L. McCaalin who 
served' :as project director and to Ms. Kay A. Adams^who helped 
develop and try out the evaluation system. Finally, special 
recognition is due to Dr. Jerry Walker, Associate Director for 
, Evaluation, foy his advice and direction- and to Mr^. Marlene 
Linton and MsJMary Schmidt for their efficient typing; of the 
manuscript. \ ^ * ^ - 

, ^ Robert E. Taylor, Director 

The Center for Vocational Education 
' The Ohio State University 
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' Introduction 

The Kentucky State Advisory Council for Vocational Educa- 
tion and 'Manpower Development and. Training recognized a need 
to increase the involvement of local and regional advisory 
committees in evaluating vo^cational programs. Although 
advisory councils in most states are asked to evaluate voca- 
tional programs, they are rarely given systematic guidance 
or appropriate tools to use. More frequently, advisory 
committees are asked to evaluate features of vocational ^ 
programs that they have limited knowledge about, such as the \- 
philosophy , objectives , management and resourced of the 
program. On the other hand, advisory Committees are typically 
not- asked to evaluate the features of programs which they 
are the most qualified to give ac;ivice about, such as the 
relevance of curriculum to current, job practices and the 

.quality of students* job-entry and employabili ty skills ^ 

The Kentucky State Advisory Council for Vocational Educa- . 
tion and Manpower Development and Training realized that 

•advisory committees were a rich ypt not fully tapped resource 
for vocati^G^al education in Kentucky, since they are able to 
provide advice which is not easily obtainable from any other ^ 
source. To better tap advisory committees' unique expertise, 
the State Advisory Council commissioned The Center for Vocational 
Education to develop an evaluation system which would guide 
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reqional anci local advisory committees in their efforts to 
provide constructive feedback for improving vocational programs. 

This rv^port has been prepared to describe the process used 
in creatirv^^, -tryinq out, and revising a set of sel f-review 
instruments and guides entitled the Kentucky Advisory Committee 
Evaluation System .(KACES). The remainder of this report 
presents a description of KACES, description of the develop- 
ment process, .iiid the conclusions and recommendations of the 
project. ' 
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Description of KACES 
KACES calls for the participation of local craft committees, 
regional advisory committees/ and the State Advisory Council 
in the evaluation of local vocational education programs 
throughout the state of Kentucky. The primary purpose of , 
KACES is to help improve Kentucky's vocational education " 
programs, not to prove whether they are good or bad. A 
secondary purpose of KACES is to increase advisory committees' 
knowledge about and involvement in vocational education so that 
they will be better able to provide advice. 

The results- of the evaluation will help ideojferi^^N^he 
greatest needs arid most outstanding strengths of the v)ocational 
programs throughout the state as viewed by adv^is^fry committees. 

will also. provide specific, practical, and meaningful 
recommendations for making program improvements-; 

KACES is a four-stage process including orientation, 
investigation, interpretation, and recommendation. Advisory 
committees' at the local, regional, and state levels will have 
a different role* to play in each stage oV the evaluation. 

■ ■ : • - 6 ^ . ■ /) 




To r ic i 1 i tat j n vo I viMiit mi t ac toss ' Ken t.ucky , KACI'iS has been ci'ivided 
into sections so that iwch Lt^vel of advisory committees (e.g., 
state, regional and local) will receive the guidance needed 
for their part in the evaluation. KACES contains seven ♦ 
evaluation guides. Thesc^ are; 

1- Organization Manual 

2- c;Oidolines for the State Advisory Council . ^ 

], Procedures for the Regional Advisory Committee 

4. C/uidelinOs for the [Regional Director of Vocational, 
education - . 

/~ * " " 

^. Procedures for the Local Craft Committee 

6- Guidc}lines fbr th^ Vocational Teach'er 

7- , Interview Guide \ 

The entire evaluation revolves around two short evaluation 
instruments which will be comple^vl by advisory committees 
at the local, regional, and state leveX^. These instrumentHS 
are ?i Profile Form and a Recommendations. Form . A one-page 
Program Prof i l e wi 1 1 provide ^' V,i sual picture of the features 
of each . local vocational program which most need to be improved. 
F^rogram Profiles will then be summarized far the region to-, 
r^r.tray needs at^that level. Regional Profiles will, in turn, 

be summarized to portray the greatest needs for the state of 

i 

Kentucky. Recommenda tipn^ and. commendations Will also be 
recorded for each local program on a Program Recommendations 
for'm. These will -then be summarized for each region* and 
then' furthe'r summarized for the. entire state. ' . 

7 ' ■" 



■l)i^j;cript ion of the I'rocoss 

Fivo .stu[>s have hron com[)loLccl in the- creation, pilot-tost, 
cind revision of KACMIS. These are: 

1. Dotorinino the nia jor area^; to be evaluated; 

2 . DevoiofT preliminary in.s truinen ts and quides for usir^)^ 
them 

3. Review arid revise the preliminary evaluation system 

4 . Try ou t the inst rumerlts and quides i n two regions of 
Kentucky 

, » 5: Prepare a final version of the KACE^ instruments and 
guides . ' . ^ ' 

The activities accomplished in each step are described in the 
following sections. 

Determine the Major Areas to be Evaluated 

A comprehensive list of evalua''tion questj-ons were prepared. 
The questions were shared with several members of the State 
Advisory Counci.1 who recommended reducing the list considerably 
They also recommended that the evaluation focus only on arqas 
which were important for advisory committees to evaluate and 
which wdre not already being evaluated through other means . 
A shorter list of 15 major evaluation questions were developed. 
(Se^e^igure 1) The importance and difficulty of answering 
these evaluation questions were rated by 83 State and Regional 

i 

Advisory Comini ttee Members and Vocational Educators at their 
joint meeting held in Lexington, Kentucky on October 23, 1975. 
The three areas selected as most importajj^t to evaluate were: 

■ 8 



! 

\ 



III v ( M 1 t 1 < w I . I I ( * o M I s < " I ( ' I I I ' ( • t iM 11 M ' n t I ) I . u ' t 1 I * I ' ; ; . 1 1 n I 

( 1 ( I ' . Ill M U I I 1 S t t V 

I . . , ) ( >l > i I I y ; k 1 I 1 : . . M« i n l( Mi t : ; I ( m\m • I ! ( i . n 1 1 i nq | • i j i ,nn 
with t } M ' < M 1 1 ry 1 1 • vt • I oTc-upa t i ( )Im I skill:; to pci i o i m t h< • 
H ) 1 1 I ( • t w 1 1 M • i I t 1 1 ( • y W( • r ( ' t I \ i i M ( M I 

I ' . i iiij ' I ( ^y . 1 1 > il I t y ( ) 1 S t Ui 1< Ml tls , Aim ' ( iul( mi I s mI > 1 c t \ \ wd 
, 1 ( )1 , In I ht ^ k 1 1 1 m rr m 1 1 > r wh i ('i^i t h« • y wi m m > t r a i m i 
M t t ' I 1 ( M\' i riq S(Mi( X) 1 

I )« -v'* * I ( ){ > IM^ ■ 1 un 1 iiM r y I ri:; t i unu mi t s miuI (iuidt-:; loi ''^^inu 'I'Iumii 

1 )!< 1 1 t 1 11", t I unit Ml t ,) Mil (}u i ii( tor" t^va 1 mm t i iiq t.h(' t lit(M • , \ r < mm 
: ' 1 « '{ ' t ( 1 M.M !Ti(»:M ] iiif h ) r t a ri were ( ir V( • 1 cjf u.m i . 

t. -M'-w Mill iM'V'iMt' t IMM^I inn nary Mat.(.'rials 

MtrT t draft nia t(.' r i a I s W(Mm..^ dpvcM c)[)f:M] , a day was s|)(MjI 
■.v!tfi Mr. Milly IknvMtMi ar^l M t- , F^iyinonci I'lilton to roco.i.V(^ tH(Mr" 
r(Mvn>aL'k and advicM^ for i.iii[>r(;v i nq tdir^ niata.MMal. Usinc] tduM r 
MUf jqt 'S 1 orif^ , t 1m.' nia t cMMa Is- were revistni and more fully deVi^lopcMl. 

Try 1 nil Llie I rist ruments and r,ui.des in Two Recjions in 'Kent ucky 

M.ui I f i( )ri V i 1 1 ( ' in Ro(]\on II and Covincjton in Hoqion VJ"I 
v;< M e * r.e It 'Ct i'd a:; tlh' t:wo i^itos for p^ilot tosfincf the preliminary 
1 ns t r MHi'Mi ts and quides. Three vis'its werc^ made to each site, 
(m t ho f irst. VM S it., reqional advisory oonimittee members were 
i n t rrnliic* 'd to tdie c^valu<\tion instruments and quides in' a three 
liour' sr-ssion. Mi^mbi^rs reacted to the materials and qave 
suqcjos 1 1 oris for imi^roviriq theni. The suqqestions of the reqional 
advisory committees wi^re used to further refine the materials. 

At th(' iM;^cf:>nd mc^etinq, five craft committees in each region 
v;tM o i n t rod wc-e'd to Llio^ systMMn and prepared .to'te^.t the materials 
duiinq a one month Lrial run. Tlie vocational proqr^ams participating 
\ roni Mad i iMCvnv i I Le wrq-e : ^ 



10 



i I , ! r I [ .11 t I ■ •< i . I u> M I I ' 'I I '1' I 1 ■ 1. 1 i 'oim ( V i I 1 ' I ' 1 : '.^ "1 w '( > I 

i ., I 1 ( ! , ( 'I ^l, h I I • >. >Mv I 1 1 (i 1 1« M M li omip. I ( I ons Aiun 

.\i,f,, • 1« •< ' h. I n 1 " . ''^ i-'ii i*i)unty V()vM( School 

1 1 1 • ; 1 1 1. ( Ml 1 1 1 1 ( ■ 1 1 1 >u I ' 1 t'l )i 1 1 1 1 V V't M M t i ( )n. I 1 Si 'Ih >c) I ^ 

u M t 1 ( WKi 1 { 'M V ] t . iin:; p.n t i c i | m t i n< i I r oiiy ( '(k' i iiu t ( )ii m o : 

\ 1 

; \ 




•\ t In r < 1 iiu M ' I 



liri'i w. IS. held h)- rjjpw^cw t lu> icr.ults ot the' pilot 



t,'..t o! t'fiM ov.iliMt loll syr.tpTii. 'All t(^n (T.ilt ^coninii 1. 1 eos 

o( >iiip I o t ( •( 1 t lio f^vMlu.it ion proc:er.:>. Kach (/ornmittco 



-M. i con i ue tv'« 1 1 n t o 1 v i evvr; with t luiont s , > | radun t. es , em[) 1 oyrT r. , 
,,n.i "o(Mt ifMi.il t(Mfh(Ms>; [)roparu'd rocorninonda t. i ons and c^oiuri(Mida ■ 



f 1 ( -n . ; lot t 1 iO v< X.M 



t I fMia I f.) r-oa !-aiu ; and I i s la^d ( li< ' ina ir7r\ iiocds 



<a I r;u a"o-/oiM» ■!! t o! t ho vooaticMial iuca fi (;n ra'oqram. ^ 

'dhsK/k o,Mu*orrnni) th^^ iua)or stronqth;; <ir^i woaknos'so:; <)\ 

.-vMliMtioM :;vst'M]i a:-; w» > 1 1 .ir, prac:d_icMl u< p f (vs 1 1 on : ; lOr inaknv; 

p,. r-MK'-'^ss ''a>".ior, h';;:-; L i ■ con:>aminq an<i moro u5-3<jiu] '.vcr*- 

;:P , t J o:'i Ml-' c-rMft t -opuii i t t o( * rnonil.n^'s and \'CM;a 1 1 on<i 1 (■dac. ilor's 
^P'.-d iM t ho I'^^'^i^ t.':',t. A sanimary oh [)a r t,\i c i [)a n t s ' oomniorida- 
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q. Difficult to'condpct intervigws 'on own time, esp4^ially^^ ^ 
difficult to' int^rvi^w other emplove^ *'v ^-i':!>^' '^l 

h. Some of the questions are hard^o ^understand _ 

y / Prepare Final Instruments and Guide , ' - ' *• 

^' ' The results of the pilot test w.ere • used, to prepare the 'fi'rjal 5 
version of the Kentucky Advisory Committee Evaluat: System. 
On preparing the final version an ^ef 'ort was made to retain, 
what participants felt were the strengths cf the materials 
while improving the weaknesses. ' - 

t 

Conclusions and Recommendations of the Project 
Kentucky Advisory Committee members in Regions II andVII 
responded very positively to the evaluation system. As a group, 
the advisory and craft commi ttees , f el t that the Kenttfcky 
Advisory Committee Evaluation System was a helpful and practical 
way to involve advisory committees in program evaluation. 
The craft committees who participated in the pilot test recom- 
mended^ that KACES-be adopted for use by advisory committees 
throughout Kentucky. , . 

Rfecommendatibhs to be considered when implementing KACES 
statewide are listed below. 

1. The role of the vocational teacher in- the evaluation 
process should be clearly defined. Several craft 
committees mentioned that the teacher should serve as^^ 
a resource but not involved in interviews or completing 
the evaluation forms. * 

2. Since most craft committees have jobs that are 'difficult 
to leave during work hours, scheduling time for inter- 
views may be sometimes difficult. Alternative procedures, 
such as telephone interviews, should be used when necessary. 
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Advisory committees should be 'encouraged to provide 
con^truttivfe critical ' feedback;^ ahd- not to jusfc praise 
the vocational program. ^ ' ^ 



/ 



Teachers must be assured thajt this is not a personnel 
evaluation. They should be/ informed how the informati^ 
will be used. / * , 



5. Aithouqh^ detailed guides have^been developed to explain 
the evaluation procedures , orientation sessions are 
a necessity, in' a^ least the f^r^t V^ar. Small group 
' ses^ionp throughout the state are needed to clarify the 
^ process and to generate ^enthusiasm.. 

6-V When the system ife implemented, advisory committees 
* ^should be reminded that evaluation is only part of their 
job. Other functions of the advisory committees should' 
be reemphasized so that they will not fall by the t 
wayside. 
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